ˆ f. Contents. Overview. Function Approximation. f ˆ : X Y. y x m. Introduction to Radial Basis Function Networks RBF

Transcription

1 Introducton to Radal Bass Functon Networks Contents Overvew he Models of Functon Aroator he Radal Bass Functon Networks RBFN s for Functon Aroaton he Proecton Matr Learnng the Kernels Bas-Varance Dlea he Effectve Nuber of Paraeters Model Selecton Introducton to Radal Bass Functon Networks Overvew RBF Lnear odels have been studed n statstcs for about 00 ears and the theor s alcable to RBF networks whch are ust one artcular te of lnear odel. However, the fashon for neural networks whch started n the d-80 has gven rse to new naes for concets alread falar to statstcans cal Alcatons of NN Functon Aroaton Pattern Classfcaton X R l f () lc N Functon Aroaton X R f () YR e-seres Forecastng ( ) (,,, ) t f t t t3 n X R Unknown Aroator X R f : X Y f f ˆ : X Y ˆ f Y R n Y R n

2 Suervsed Learnng Neural Networks as Unversal Aroators,, Unknown Functon Neural Network ˆ + + e Feedforward neural networks wth a sngle hdden laer of sgodal unts are caable of aroatng unforl an contnuous ultvarate functon, to an desred degree of accurac. Lke feedforward neural networks wth a sngle hdden laer of sgodal unts, t can be shown that RBF networks are unversal aroators. Statstcs vs. Neural Networks Introducton to Radal Bass Functon Networks Statstcs Model Estaton Regresson Interolaton Observatons Paraeters Indeendent varables Deendent varables Rdge regresson Neural Networks Network Learnng Suervsed learnng Generalzaton ranng set (snatc) Weghts Inuts Oututs Weght deca he Model of Functon Aroator Lnear Models Lnear Models f () w ( ) f Weghts () w ( ) Fed Bass Functons Hdden Unts Inuts Outut Unts Lnearl weghted outut w w w = n Decooston Feature Etracton ransforaton Feature Vectors

3 Lnear Models Hdden Unts Inuts = Outut Unts Lnearl weghted outut w w w n f () w ( ) Decooston Feature Etracton ransforaton Feature Vectors Eale Lnear Models Polnoal f () w Fourer Seres f () w e k k k ( ), 0,,, k k0 k 0 f () w ( ) ( ) e, 0,,, Sngle-Laer Percetrons as Unversal Aroators f () w ( ) Radal Bass Functon Networks as Unversal Aroators f () w ( ) Hdden Unts w w w Wth suffcent nuber of sgodal unts, t can be a unversal aroator. Hdden Unts w w w Wth suffcent nuber of radal-bass-functon unts, t can also be a unversal aroator. = n = n Non-Lnear Models f () w ( ) Introducton to Radal Bass Functon Networks f Weghts () w ( ) Adusted b the Learnng rocess he Radal Bass Functon Networks 3

4 f () w ( ) Radal Bass Functons hree araeters for a radal functon: Center Dstance Measure Shae r = ()= ( ) cal Radal Functons Gaussan r r e 0 and r Hard-Multquadratc (97) r r c c c 0 and r Inverse Multquadratc r c r c c 0 and r r r e 0 and r Gaussan Bass Functon (=0.5,.0,.5) r c r c c 0 and r Inverse Multquadratc c=5 c=4 c=3 c= c= Bass { : =,, } s `near orthogonal. Most General RBF ( μ) ( μ) μ μ μ 3 Proertes of RBF s On-Center, Off Surround Analoges wth localzed recetve felds found n several bologcal structures, e.g., vsual corte; ganglon cells 4

5 As a functon f() aroator he oolog of RBF As a attern classfer. he oolog of RBF Outut Unts Interolaton Outut Unts Classes Hdden Unts Proecton Hdden Unts Subclasses Inuts n Feature Vectors Inuts n Feature Vectors Introducton to Radal Bass Functon Networks RBFN s for Functon Aroaton Radal Bass Functon Networks Radal bass functon (RBF) networks are feedforward networks traned usng a suervsed tranng algorth. he actvaton functon s selected fro a class of functons called bass functons. he usuall tran uch faster than BP. he are less suscetble to robles wth nonstatonar nuts Radal Bass Functon Networks he dea Poularzed b Broohead and Lowe (988), and Mood and Darken (989), RBF networks have roven to be a useful neural network archtecture. he aor dfference between RBF and BP s the behavor of the sngle hdden laer. Rather than usng the sgodal or S-shaed actvaton functon as n BP, the hdden unts n RBF networks use a Gaussan or soe other bass kernel functon. Unknown Functon to Aroate ranng Data 5

6 he dea he dea w f( ) ( ) Unknown Functon to Aroate Functon Learned ranng Data Bass Functons (Kernels) Bass Functons (Kernels) he dea w f( ) ( ) he dea w f( ) ( ) Nontranng Sale Functon Learned Nontranng Sale Functon Learned Bass Functons (Kernels) Radal Bass Functon Networks as Unversal Aroators ranng set, Goal ( k ) ( k f ) for all k n SSE k f k w k f ( ) w ( ) w w w = n Learn the Otal Weght Vector ranng set, Goal ( k ) ( k f ) for all k n SSE k f k w k f ( ) w ( ) w w w = n 6

7 Regularzaton ranng set, Goal ( k ) ( k f ) for all k n SSE C k f w k k w w 0 If regularzaton s unneeded, set 0 Learn the Otal Weght Vector Mnze C w f ( ) ( ) w f k C f 0 w f w k w k k ( ) ( ) f w k f w ( ) w ( ) k k f w f ( ) ( ) ( ) w ( ) Learn the Otal Weght Vector f φ f w φ,, Learn the Otal Weght Vector φ f w φ,, Defne () ( ),, f,, f () ( ),, φ f () ( ) f w k k φ f w φ w φ f φ f w φ φ Defne Φ f Λw Φ φ, φ,, φ Λ w w, w,, w Φ Learn the Otal Weght Vector () () wk k () () () f k () () w f wk k f () () () w k ( ) f ( ) ( ) ( ) w ( P) wk k k Φ Φw Λw Φ f Λw f Φ Φ ( ) w ( ) Φw Φ Φw Λw Φ Learn the Otal Weght Vector Φ ΦΛ w Λ Φ w Φ Φ Φ A Φ Φ: A : Desgn Matr Varance Matr 7

8 Introducton to Radal Bass Functon Networks w A Φ he Ercal-Error Vector f Φw he Proecton Matr Unknown Functon w w φ φ w φ n n w A Φ he Ercal-Error Vector f Φw Error Vector w w w Unknown φ φ e f Φw ΦA Functon I ΦA Φ P n Φ φ n Su-Squared-Error Error Vector A ΦΦΛ e P SSE f k If =0, the RBFN s learnng algorth s to nze SSE (MSE). Φ φ, φ,, φ PI ΦA Φ k ( ) P P P he Proecton Matr Error Vector SSE P A ΦΦΛ e P Λ0 Φ φ, φ,, φ PI ΦA Φ e san ( φ, φ,, φ) P( P) Pe e P P Introducton to Radal Bass Functon Networks Learnng the Kernels 8

9 RBFN s as Unversal Aroators What to Learn? l w w w w l w l w ll ranng set, k Kernels μ ( ) e l w w w w l w l w ll Weghts w s Centers s of s Wdths s of s Nuber of s Model Selecton n n l f w One-Stage Learnng ( k ) ( k ) ( k ) w f l μ w f μ μ l w f 3 3 l f w he sultaneous udates of all three sets of araeters a be sutable One-Stage Learnng for non-statonar envronents or onlne settng. ( k ) ( k ) ( k ) w f l μ w f μ μ l w f 3 3 wo-stage ranng ran the Kernels l Ste Deternes w s. w w w w l w l w ll E.g., usng batch-learnng. n Ste Deternes Centers s of s. Wdths s of s. Nuber of s. 9

10 Unsuervsed ranng Methods Subset Selecton Rando Subset Selecton Forward Selecton Backward Elnaton Clusterng Algorths KMEANS LVQ Mture Models GMM Subset Selecton Rando Subset Selecton Randol choosng a subset of onts fro tranng set Senstve to the ntall chosen onts. Usng soe adatve technques to tune Centers Wdths #onts Clusterng Algorths Clusterng Algorths 0

11 Clusterng Algorths Clusterng Algorths 4 3 Introducton to Radal Bass Functon Networks Bas-Varance Dlea Goal Revst Ultate Goal Generalzaton Mnze Predcton Error Goal of Our Learnng Procedure Mnze Ercal Error Badness of Ft Underfttng A odel (e.g., network) that s not suffcentl cole can fal to detect full the sgnal n a colcated data set, leadng to underfttng. Produces ecessve bas n the oututs. Overfttng A odel (e.g., network) that s too cole a ft the nose, not ust the sgnal, leadng to overfttng. Produces ecessve varance n the oututs. Underfttng/Overfttng Avodance Model selecton Jtterng Earl stong Weght deca Regularzaton Rdge Regresson Baesan learnng Cobnng networks

12 Best Wa to Avod Overfttng Badness of Ft Use lots of tranng data, e.g., 30 tes as an tranng cases as there are weghts n the network. for nose-free data, 5 tes as an tranng cases as weghts a be suffcent. Don t arbtrarl reduce the nuber of weghts for fear of underfttng. Underft Overft Badness of Ft However, t's not reall a dlea. Bas-Varance Dlea Underft Overft Underft Overft Large bas Sall varance Sall bas Large varance It's not reall a dlea. Bas-Varance Dlea Bas-Varance Dlea More on overfttng Underft Easl lead to redctons that are far beond the range of the tranng data. Produce wld redctons n ultlaer ercetrons even Large wth nose-free bas data. Sall varance Overft Sall bas Large varance ore underft Bas Varance ft ore overft

13 he ean of the bas=? he varance of the bas=? Bas-Varance Dlea he ean of the bas=? he varance of the bas=? Bas-Varance Dlea nose bas bas bas he true odel Varance nose bas he true odel Soluton obtaned wth tranng set. Soluton obtaned wth tranng set 3. Soluton obtaned wth tranng set. Sets of functons E.g., deend on # hdden nodes used. Sets of functons E.g., deend on # hdden nodes used. Reduce the effectve nuber of araeters. Reduce the nuber of hdden nodes. Model Selecton Goal: n E ( ) f ( ) Bas-Varance Dlea Varance nose bas he true odel ( ) g( ) nose he true odel g() bas f () Sets of functons E.g., deend on # hdden nodes used. Sets of functons E.g., deend on # hdden nodes used. Goal: n E ( ) f ( ) Bas-Varance Dlea Goal: n E ( ) f ( ) n Eg( ) f ( ) ( ) ( ) E g( ) f ( ) E f ( ) ( ) ( ) ( ) E g f g f ( ) ( ) ( ) ( ) E g f E g f E 0 constant ( ) ( ) ( ) ( ) E f E E g f Bas-Varance Dlea Eg( ) f ( ) E g( ) E[ f ( )] E[ f ( )] f ( ) Eg( ) E[ f ( )] f ( ) E[ f ( )] g( ) E[ f ( )] f ( ) E[ f ( )] Eg( ) Ef [ ( )] E f( ) Ef [ ( )] E g( ) E[ f ( )] f ( ) E[ f ( ) ] 0 E g( ) E[ f ( )] f( ) E[ f ( )] Eg( ) f ( ) Eg( )E[ f ( )] EEf [ ( )] f ( ) EE[ f ( )] E[ f ( )] Eg( ) E f ( ) Eg( ) E[ f ( )] Ef [ ( )] E f () E[ f ( )] E[ f ( )] 0 3

14 Goal: n E ( ) f ( ) Bas-Varance Dlea ( ) ( ) ( ) ( ) E f E E g f E Eg( ) Ef [ ( )] E f ( ) Ef [ ( )] nose bas varance Goal: n E ( ) f ( ) Model Colet vs. Bas-Varance ( ) ( ) ( ) ( ) E f E E g f E Eg( ) Ef [ ( )] E f ( ) Ef [ ( )] nose bas varance Cannot be nzed Mnze both bas and varance Model Colet (Caact) Goal: n E ( ) f ( ) Bas-Varance Dlea ( ) ( ) ( ) ( ) E f E E g f E Eg( ) Ef [ ( )] E f ( ) Ef [ ( )] Eale (Polnoal Fts) e( 6 ) nose bas varance Model Colet (Caact) Eale (Polnoal Fts) Eale (Polnoal Fts) Degree Degree 5 Degree 0 Degree 5 4

15 Introducton to Radal Bass Functon Networks he Effectve Nuber of Paraeters Varance Estaton Mean ˆ In general, not avalable. Varance Varance Estaton Sle Lnear Regresson Mean ˆ ˆ s ˆ Varance 0 ~ N(0, ) 0 Loss degree of freedo Mnze ˆ Sle Lnear Regresson SSE Mnze ˆ Mean Squared Error (MSE) SSE 0 ~ N(0, ) ŷ ˆ ˆ 0 0 ~ N(0, ) ŷ ˆ ˆ 0 ˆ SSE MSE Loss degrees of freedo 5

16 Varance Estaton SSE ˆ MSE Loss degrees of freedo : #araeters of the odel he Nuber of Paraeters f ( ) w ( ) w w w = n #degrees of freedo: he Effectve Nuber of Paraeters () Facts: trace ( AB) trace ( A) trace ( B) trace ( AB) trace ( BA) he Effectve Nuber of Paraeters () f ( ) w ( ) he roecton Matr w w w = n PI ΦA Φ A ΦΦΛ Λ 0 trace( P) Pf) trace ( ) P trace I ΦA Φ trace ΦA Φ trace Φ Φ Φ Φ trace Φ Φ Φ Φ trace I he roecton Matr PI ΦA Φ A ΦΦΛ Λ 0 trace( P) he effectve nuber of araeters: trace( P) Regularzaton he effectve nuber of araeters: trace ( P) Regularzaton Cost Eral Error f k w SSE k Penalze odels wth large weghts Model s enal t w Cost Eral Error f k w SSE k Penalze odels wth large weghts Wthout enalt ( Model =0), there are degrees s of freedo to nze enal SSE t(cost). he effectve nuber of araeters =. w 6

17 he effectve nuber of araeters: ( ) he effectve nuber of araeters: trace ( P) trace P Regularzaton Varance Estaton Cost Eral Error f k w SSE k Penalze odels wth large weghts Model s enal t Wth enalt ( >0), the lbert to nze SSE wll be reduced. he effectve nuber of araeters <. w ˆ MSE SSE Loss degrees of freedo Varance Estaton ˆ he effectve nuber of araeters: trace ( P) SSE MSE trace ( P) Introducton to Radal Bass Functon Networks Model Selecton Model Selecton Goal Choose the fttest odel Crtera Least redcton error Man ools (Estate Model Ftness) Cross valdaton Proecton atr Methods Weght deca (Rdge regresson) Prunng and Growng RBFN s Ercal Error vs. Model Ftness Ultate Goal Generalzaton Mnze Predcton Error Goal of Our Learnng Procedure Mnze Ercal Error (MSE) Mnze Predcton Error 7

18 Estatng Predcton Error When ou have lent of data use ndeendent test sets E.g., use the sae tranng set to tran dfferent odels, and choose the best odel b coarng on the test set. When data s scarce, use Cross-Valdaton Bootstra Cross Valdaton Slest and ost wdel used ethod for estatng redcton error. Partton the orgnal set nto several dfferent was and to coute an average score over the dfferent arttons, e.g., K-fold Cross-Valdaton Leave-One-Out Cross-Valdaton Generalze Cross-Valdaton K-Fold CV Slt the set, sa, D of avalable nut-outut atterns nto k utuall eclusve subsets, sa D, D,, D k. K-Fold CV Avalable Data ran and test the learnng algorth k tes, each te t s traned on D\D and tested on D. K-Fold CV est Set ranng Set A secal case of k-fold CV. Leave-One-Out CV D D D 3 Avalable. Data.. D k D D D 3... D k D D D 3... D k D D D 3... D k D D D 3... D k Estate Slt the avalable nut-outut atterns nto a tranng set of sze and a test set of sze. Average the squared error on the left-out attern over the ossble was of artton. 8

19 A secal case of k-fold CV. Error Varance Predcted b LOO D (, ): k,,, k D D\ (, ) f k,, Functon learned usng D as tranng set. Avalable nut-outut atterns. ranng sets of LOO. he estate for the varance of redcton error usng LOO: ˆ ( ) LOO f Error-square for the left-out eleent. A secal case of k-fold CV. Error Varance Gven a odel, Predcted the functon wth b least LOO D (, ): k,,, k D D\ (, ) f k ercal error for D.,, Functon learned usng D as tranng set. Avalable nut-outut atterns. ranng sets of LOO. As an nde of odel s ftness. he estate We want for to fnd the a varance odel also of redcton nze ths. error usng LOO: ˆ ( ) LOO f Error-square for the left-out eleent. A secal case of k-fold CV. Error Varance Predcted b LOO D (, ): k,,, k D D\ (, ) f k,, Functon learned usng D as tranng set. Avalable nut-outut atterns. ranng sets of LOO. he estate for the varance of redcton error usng LOO: ˆ ( ) LOO f Error-square for the left-out eleent. Error Varance Predcted b LOO ˆLOO ˆ P( dag( P)) Pˆ ˆ ( ) LOO f Error-square for the left-out eleent. ˆ LOO ˆ P( dag( P)) Pˆ Generalzed Cross-Valdaton More Crtera Based on CV trace ( P) dag ( P) I ˆ GCV ˆ ˆ trace ( P) P ˆ ˆ ( ) P ˆ GCV ˆ ˆ ( ) P ˆ P ˆ ˆUEV ˆ ˆ ˆFPE GCV (Generalzed CV) UEV (Unbased estate of varance) P FPE ln( ) ˆ P ˆ ˆ BIC (Fnal Predcton Error) Akake s Inforaton Crteron BIC (Baesan Inforaton Crtero) 9

20 ˆ ˆ ˆ ˆ UEV FPE GCV BIC More Crtera Based on CV ˆ ˆ ˆ ˆ UEV FPE GCV BIC More Crtera Based on CV ˆ GCV ˆ ˆ ( ) P ˆ P ˆ ˆUEV ˆ P ˆ ˆFPE ˆUEV ln( ) ˆ P ˆ ˆ BIC ˆUEV ln( ) ˆUEV ˆ GCV ˆ ˆ ( ) P ˆ P ˆ ˆUEV ˆ P ˆ ˆFPE ˆUEV ln( ) ˆ P ˆ ˆ BIC ˆUEV P? ln( ) ˆUEV Standard Rdge Regresson, Regularzaton, Standard Rdge Regresson, Regularzaton, SSE Penalze odels wth large weghts SSE Penalze odels wth large weghts Cost Eral Error f k w k Model s enal t w Cost Eral Error f k w k Model s enal t w w n Soluton Revew w A Φ A Φ ΦI C w w k PI ΦA Φ Used to coute odel selecton crtera Eale Wdth of RBF r = 0.5 ( ) ( ) e ~ N(0,0. ) 50 0

21 Eale Wdth of RBF r = 0.5 ( ) ( ) e ~ N(0,0. ) 50 ˆ ˆ ˆ ˆ UEV FPE GCV BIC Eale Wdth of RBF r = 0.5 ( ) ( ) e ~ N(0,0. ) 50 ˆ ˆ ˆ ˆ UEV FPE GCV BIC Otzng the Regularzaton Paraeter Re-Estaton Forula P trace A ˆ A ˆ w A wtrace ( P) Local Rdge Regresson Re-Estaton Forula P trace A ˆ A ˆ w A wtrace ( P) Eale trace( P) sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. ) Eale trace( P) sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. )

22 Eale sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. ) Eale sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. ) P trace A ˆ A ˆ w A wtrace ( P) here are two local-na. P trace A ˆ A ˆ w A wtrace ( P) here are two local-na. Usng the about re-estaton forula, t wll be stuck at the nearest local nu. ˆ(0) 0.0 hat s, the soluton deends on the ntal settng. l ˆ( t ) 0. t Eale P trace A ˆ A ˆ w A wtrace ( P) ˆ(0) 0 5 sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. ) here are two local-na. Eale RMSE: Root Mean Squared Error In real case, t s not avalable. sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. ) l ˆ() t. 0 t 4 Eale sn( ) 50 r 0.05 Wdth of RBF ~ N(0,0. ) RMSE: Root Mean Squared Error In real case, t s not avalable. Local Rdge Regresson Standard Rdge Regresson n n C w w k Local Rdge Regresson C w w k

23 Local Rdge Regresson Standard Rdge Regresson n C les that () can be reoved. w w k Local Rdge Regresson n C w w k he Solutons w A Φ Lnear Regresson A Φ Φ A Φ ΦI A Φ ΦΛ PI ΦA Φ Used to coute odel selecton crtera Standard Rdge Regresson Local Rdge Regresson ˆ P ˆ ˆ GCV Otzng the trace ( P) Regularzaton Paraeters Otzng the Regularzaton Paraeters Increental Oeraton P: he current roecton Matr. P : he roecton Matr obtaned b reovng (). Solve ˆGCV 0 Subect to 0 ˆ GCV ˆ ˆ trace ( P) P PP P φ φ P φ P φ PP P φ φ P φ P φ Otzng the Regularzaton Paraeters Otzng the Regularzaton Paraeters Solve ˆGCV 0 Subect to 0 a P b Pφ Pφ ( ) c φ P φ Pφ trace ( P ) φ P φ f a b cb ˆ f φ Pφ ba and a b cb 0 f φ Pφ ba and a b c b c b b a φ Pφ f φpφ b a Solve ˆGCV 0 Subect to 0 a P b Pφ Pφ ( ) Reove () c φ P φ Pφ trace ( P ) φ P φ f a b cb ˆ f φ Pφ ba and a b cb 0 f φ Pφ ba and a b c b c b b a φ Pφ f φpφ b a 3

24 he Algorth Intalze s. e.g., erforng standard rdge regresson. Reeat the followng untl GCV converges: Randol select and coute Perfor local rdge regresson If GCV reduce & reove () ˆ ˆ 4